Augmented optical imaging system for use in medical procedures

ABSTRACT

An optical imaging system for imaging a target during a medical procedure is disclosed. The optical imaging system includes: a first camera for capturing a first image of the target; a second wide-field camera for capturing a second image of the target; at least one path folding mirror disposed in an optical path between the target and a lens of the second camera; and a processing unit for receiving the first image and the second image, the processor being configured to: apply an image transform to one of the first image and the second wide-field image; and combine the transformed image with the other one of the images to produce a stereoscopic image of the target.

TECHNICAL FIELD

The present disclosure relates to medical imaging and, in particular, tooptical imaging systems suitable for use in image-guided medicalprocedures.

BACKGROUND

Digital microscopes support advanced visualization during medicalprocedures. For example, digital surgical microscopes provide magnifiedviews of anatomical structures during a surgery. Digital microscopes useoptics and digital (e.g. CCD-based) cameras to capture images inreal-time and output the images to displays for viewing by a surgeon,operator, etc.

In image-guided medical applications, such as surgery or diagnosticimaging, accurate three-dimensional (3-D) visualization of patientanatomy and surgical tools is crucial. It would be desirable to providelightweight digital microscope solutions that support accurate 3-Dvisualization.

BRIEF DESCRIPTION OF DRAWINGS

Reference will now be made, by way of example, to the accompanyingdrawings which show example embodiments of the present application andin which:

FIG. 1 shows an example navigation system to support image-guidedsurgery;

FIG. 2 illustrates components of an example navigation system;

FIG. 3 is a block diagram illustrating an example control and processingsystem which may be used in the example navigation system of FIGS. 1 and2;

FIG. 4A shows the use of an example optical imaging system during amedical procedure;

FIG. 4B is a block diagram illustrating components of an example opticalimaging system 500;

FIGS. 5A-5E show different views of an example augmented optical imagingsystem;

FIGS. 6A-6B show different perspective views of an example module foraugmenting an optical imaging system;

FIGS. 7A-7D show optical paths for the cameras of the augmented opticalimaging system of FIGS. 5A-5E;

FIG. 8 is a partial side cross-sectional view of the augmented opticalimaging system mounted on a positioning system;

FIG. 9 shows a perspective view of another example augmented opticalimaging system; and

FIG. 10 shows, in flowchart form, an example method of generating astereoscopic image of a target using the augmented optical imagingsystem of FIGS. 5A-5E.

Like reference numerals are used in the drawings to denote like elementsand features.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

In one aspect, the present disclosure describes an optical imagingsystem for imaging a target during a medical procedure. The opticalimaging system includes: a first camera for capturing a first image ofthe target; a second wide-field camera for capturing a second image ofthe target; at least one path folding mirror disposed in an optical pathbetween the target and a lens of the second camera; and a processingunit for receiving the first image and the second image, the processingunit being configured to: apply an image transform to one of the firstimage and the second image; and combine the transformed image with theother one of the images to produce a stereoscopic image of the target.

In some implementations, the first camera, the second camera, and the atleast one path folding mirror may be housed within a single housing.

In some implementations, the second camera and the at least one pathfolding mirror may be included in an add-on module for mounting to thefirst camera.

In some implementations, the at least one path folding mirror maycomprise a first mirror and a second mirror that are selectivelypositioned based on a position of the lens of the second camera, thefirst mirror and the second mirror being angled with respect to eachother.

In some implementations, the first mirror may be selectively positionedand angled with respect to the target so as to reflect an image of thetarget to the second mirror, and the second mirror may be selectivelypositioned and angled so as to reflect the image of the target from thefirst mirror to the lens of the second camera.

In some implementations, the first camera and the second camera may bepositioned such that an optical axis of the first camera is co-planarwith the optical axis of the second camera.

In some implementations, the image transform may be a homographictransform.

In some implementations, the processing unit may be further configuredto: determine a working distance between the target and an aperture ofthe optical imaging system; and determine the image transform based onthe working distance.

In some implementations, the optical imaging system may be configured tobe mountable onto a moveable support structure.

In some implementations, the optical imaging system may further comprisea support connector to enable the optical imaging system to be removablymounted onto the moveable support structure.

In some implementations, the moveable support structure may comprise oneof a robotic arm, a manually-operated support arm, or a moveable supportframe.

In some implementations, the optical imaging system may further includea manual release button that, when actuated, enables the optical imagingsystem to be positioned manually.

In some implementations, the processing unit may be responsive tocontrol input received via a user interface.

In some implementations, the optical imaging system may further includeone or more light sources.

In some implementations, the second camera may have at least one offixed zoom optics or fixed focus optics.

In some implementations, the second camera may be fixedly coupled to thefirst camera.

In another aspect, the present disclosure describes a method ofgenerating a stereoscopic image of a target in a medical procedure usingan optical imaging system. The method includes: receiving, from a firstcamera of the optical imaging system, a first image of the target;receiving, from a second camera of the optical imaging system, a secondimage of the target; applying an image transform to one of the firstimage and the second image; and combining the transformed image with theother one of the images to produce the stereoscopic image of the target.

In some implementations, the method may further include determining aworking distance between the target and an aperture of the opticalimaging system; and determining the image transform based on the workingdistance.

In some implementations, the method may further include selecting thefirst homographic transform from a plurality of homographic transforms,wherein the selecting comprises: for each of the plurality ofhomographic transforms: applying the homographic transform to the secondimage; computing an image correspondence metric between the transformedsecond image and the first camera, and selecting the homographictransform having a highest value of image correspondence metric from theplurality of homographic transforms as the first homographic transform.

Other example embodiments of the present disclosure will be apparent tothose of ordinary skill in the art from a review of the followingdetailed descriptions in conjunction with the drawings.

In the present application, the phrase “access port” is intended torefer to a cannula, a conduit, sheath, port, tube, or other structurethat is insertable into a subject, in order to provide access tointernal tissue, organs, or other biological substances. In someembodiments, an access port may directly expose internal tissue, forexample, via an opening or aperture at a distal end thereof, and/or viaan opening or aperture at an intermediate location along a lengththereof. In other embodiments, an access port may provide indirectaccess, via one or more surfaces that are transparent, or partiallytransparent, to one or more forms of energy or radiation, such as, butnot limited to, electromagnetic waves and acoustic waves.

In the present application, the term “intraoperative” is intended torefer to an action, process, method, event, or step that occurs or iscarried out during at least a portion of a medical procedure.Intraoperative, as defined herein, is not limited to surgicalprocedures, and may refer to other types of medical procedures, such asdiagnostic and therapeutic procedures.

In the present application, the term “and/or” is intended to cover allpossible combinations and sub-combinations of the listed elements,including any one of the listed elements alone, any sub-combination, orall of the elements, and without necessarily excluding additionalelements.

In the present application, the phrase “at least one of . . . or . . . ”is intended to cover any one or more of the listed elements, includingany one of the listed elements alone, any sub-combination, or all of theelements, without necessarily excluding any additional elements, andwithout necessarily requiring all of the elements.

Various medical procedures, such as surgery and diagnostic imaging,employ digital microscopes, which provide magnified views of anatomicalstructures in real-time. Typically, digital microscope systemsincorporate a single main camera (or video-scope) for capturing imageswhich are output to a display for viewing by a surgeon or operator. Themain camera provides a single feed of video data, and the frames of thevideo feed are presented as two-dimensional images. As a result, 3-Dvisualization and, more specifically, depth perception may be absent inthese limited digital microscope systems.

In order to generate 3-D visualization, a second camera may be added toa digital microscope. The images from the two cameras can be combined toproduce stereoscopic views of a surgical site. One of the challenges inproviding a 3-D capable digital microscope is integrating two camerassuch that the microscope maintains a minimal profile in the operativefield. A simplistic arrangement of the two cameras side-by-side mayrender the microscope bulky and may result in significant obstruction ofthe surgeon's view. A small footprint for the camera modules of thedigital microscope offers a large working area for the surgeon.

Furthermore, the size of the cameras and optics may prevent the twocameras of the digital microscope from being arranged close to eachother. In particular, there may be physical restrictions to controllingthe spacing between the optical paths of the two cameras. This canresult in undesirable disparity of images from the cameras and, as aconsequence, less successful or comfortable 3-D visualizationexperience.

The present disclosure provides an augmented optical imaging system foruse in medical applications. The disclosed optical imaging system may,for example, be implemented as part of a digital microscope. The systememploys a pair of cameras, including a primary camera and an outriggercamera, for imaging a target during a medical procedure. The system alsoincludes at least one path folding mirror which is selectivelypositioned between the target and a lens of the outrigger camera. Thepath folding mirrors allow the optical path of the outrigger camera tobe manipulated such that the separate optical paths of the two camerasare substantially parallel to each other near the target. The systemprovides a 3-D visualization of the target by combining video/imageframes from the two cameras to produce stereoscopic images of thetarget.

The present disclosure also provides an optics module for extending thefunctionalities of a digital microscope system. The disclosed opticsmodule may be an add-on component to an existing optical imaging device,such as a digital microscope. The module includes an outrigger cameraand at least one path folding mirror. The path folding mirrors aredisposed in an optical path between a lens of the outrigger camera and atarget being imaged. The module is configured to be connected to theoptical imaging device. For example, the module may define a chamber forreceiving a primary camera (e.g. video-scope) of the optical imagingdevice such that both the primary camera and the outrigger camera aredirected towards the target when the module is secured to the opticalimaging device. With a minimal profile in the working field, thedisclosed optics module allows the combined optical imaging system toproduce 3-D visualization of a target.

Reference is first made to FIG. 1, which shows an example navigationsystem 200. The example navigation system 200 may be used to supportimage-guided surgery. As shown in FIG. 1, a surgeon 201 conducts asurgery on a patient 202 in an operating room environment. A medicalnavigation system 205 may include an equipment tower, tracking system,displays, and tracked instruments to assist the surgeon 201 during aprocedure. An operator 203 may also be present to operate, control, andprovide assistance for the medical navigation system 205.

FIG. 2 shows components of an example medical navigation system 205. Thedisclosed augmented optical imaging system may be used in the context ofthe medical navigation system 205. The medical navigation system 205 mayinclude one or more displays 206, 211 for displaying video images, anequipment tower 207, and a positioning system 208, such as a medicalarm, which may support an optical imaging system 500. One or more of thedisplays 206, 211 may include a touch-sensitive display for receivingtouch input. The equipment tower 207 may be mounted on a frame, such asa rack or cart, and may contain a power supply and a computer/controllerthat may execute planning software, navigation software, and/or othersoftware to manage the positioning system 208. In some examples, theequipment tower 207 may be a single tower configuration operating withdual displays 206, 211; however, other configurations (e.g. dual tower,single display etc.) may also exist.

A portion of the patient's anatomy may be held in place by a holder. Forexample, as shown in FIG. 2, the patient's head and brain may be held inplace by a head holder 217. An access port 12 and associated introducer210 may be inserted into the head, to provide access to a surgical sitein the head. The optical imaging system 500 may be used to view down theaccess port 12 at a sufficient magnification to allow for enhancedvisibility down the access port 12. The output of the optical imagingsystem 500 may be received by one or more computers or controllers togenerate a view that may be depicted on a visual display (e.g. one ormore displays 206, 211).

In some examples, the navigation system 205 may include a trackedpointer 222. The tracked pointer 222, which may include markers 212 toenable tracking by a tracking camera 213, may be used to identify points(e.g. fiducial points) on a patient. An operator, typically a nurse orthe surgeon 201, may use the tracked pointer 222 to identify thelocation of points on the patient 202, in order to register the locationof selected points on the patient 202 in the navigation system 205. Insome embodiments, a guided robotic system with closed loop control maybe used as a proxy for human interaction. Guidance to the robotic systemmay be provided by any combination of input sources such as imageanalysis, tracking of objects in the operating room using markers placedon various objects of interest, or any other suitable robotic systemguidance techniques.

Fiducial markers 212 may be connected to the introducer 210 for trackingby the tracking camera 213, which may provide positional information ofthe introducer 210 from the navigation system 205. In some examples, thefiducial markers 212 may be alternatively or additionally attached tothe access port 12. In some examples, the tracking camera 213 may be a3-D infrared optical tracking stereo camera. In some other examples, thetracking camera 213 may be an electromagnetic system (not shown), suchas a field transmitter that may use one or more receiver coils locatedon the tool(s) to be tracked. A known profile of the electromagneticfield and known position of receiver coil(s) relative to each other maybe used to infer the location of the tracked tool(s) using the inducedsignals and their phases in each of the receiver coils.

Location data of the positioning system 208 and/or access port 12 may bedetermined by the tracking camera 213 by detection of the fiducialmarkers 212 placed on or otherwise in fixed relation (e.g. in rigidconnection) to any of the positioning system 208, the access port 12,the introducer 210, the tracked pointer 222 and/or other trackedinstruments. The fiducial marker(s) 212 may be active or passivemarkers. A display 206, 2011 may provide an output of the computed dataof the navigation system 205. In some examples, the output provided bythe display 206, 211 may include axial, sagittal, and coronal views ofpatient anatomy as part of a multi-view output.

The active or passive fiducial markers 212 may be placed on tools (e.g.the access port 12 and/or the optical imaging system 500) to be tracked,to determine the location and orientation of these tools using thetracking camera 213 and navigation system 205. The markers 212 may becaptured by a stereo camera of the tracking system to give identifiablepoints for tracking the tools. A tracked tool may be defined by agrouping of markers 212, which may define a rigid body to the trackingsystem. This may in turn be used to determine the position and/ororientation in 3-D of a tracked tool in a virtual space. The positionand orientation of the tracked tool in 3-D may be tracked in six degreesof freedom (e.g. x, y, z coordinates and pitch, yaw, roll rotations), infive degrees of freedom (e.g. x, y, z, coordinate and two degrees offree rotation), but preferably tracked in at least three degrees offreedom (e.g. tracking the position of the tip of a tool in at least x,y, z coordinates). In typical use with navigation systems, at leastthree markers 212 are provided on a tracked tool to define the tool invirtual space; however, it is known to be advantageous for four or moremarkers 212 to be used.

Camera images capturing the markers 212 may be logged and tracked, by,for example, a closed circuit television (CCTV) camera. The markers 212may be selected to enable or assist in segmentation in the capturedimages. For example, infrared (IR)-reflecting markers and an IR lightsource from the direction of the camera may be used. In some examples,the spatial position and orientation of the tracked tool and/or theactual and desired position and orientation of the positioning system208 may be determined by optical detection using a camera. The opticaldetection may be done using an optical camera, rendering the markers 212optically visible.

In some examples, the markers 212 (e.g. reflectospheres) may be used incombination with a suitable tracking system, to determine the spatialpositioning position of the tracked tools within the operating theatre.Different tools and/or targets may be provided with respect to sets ofmarkers 212 in different configurations. Differentiation of thedifferent tools and/or targets and their corresponding virtual volumesmay be possible based on the specification configuration and/ororientation of the different sets of markers 212 relative to oneanother, enabling each such tool and/or target to have a distinctindividual identity within the navigation system 205. The individualidentifiers may provide information to the system, such as informationrelating to the size and/or shape of the tool within the system. Theidentifier may also provide additional information such as the tool'scentral point or the tool's central axis, among other information. Thevirtual tool may also be determinable from a database of tools stored inor provided to the navigation system 205. The markers 212 may be trackedrelative to a reference point or reference object in the operating room,such as the patient 202.

In some examples, the markers 212 may include printed or 3-D designsthat may be used for detection by an auxiliary camera, such as awide-field camera (not shown) and/or the optical imaging system 500.Printed markers may also be used as a calibration pattern, for exampleto provide distance information (e.g. 3-D distance information) to anoptical detector. Printed identification markers may include designssuch as concentric circles with different ring spacing and/or differenttypes of bar codes, among other designs. In some examples, in additionto or in place of using markers 212, the contours of known objects (e.g.the side of the access port 12) could be captured by and identifiedusing optical imaging devices and the tracking system.

A guide clamp 218 (or more generally a guide) for holding the accessport 12 may be provided. The guide clamp 218 may allow the access port12 to be held at a fixed position and orientation while freeing up thesurgeon's hands. An articulated arm 219 may be provided to hold theguide clamp 218. The articulated arm 219 may have up to six degrees offreedom to position the guide clamp 218. The articulated arm 219 may belockable to fix its position and orientation, once a desired position isachieved. The articulated arm 219 may be attached or attachable to apoint based on the patient head holder 217, or another suitable point(e.g. on another patient support, such as on the surgical bed), toensure that when locked in place, the guide clamp 218 does not moverelative to the patient's head.

In a surgical operating room/theatre, setup of a navigation system maybe relatively complicated; there may be many pieces of equipmentassociated with the surgical procedure, as well as elements of thenavigation system 205. Further, setup time typically increases as moreequipment is added. To assist in addressing this, the navigation system205 may include two additional wide-field cameras to enable videooverlay information. Video overlay information can then be inserted intodisplayed images, such as images displayed on one or more of thedisplays 206, 211. The overlay information may illustrate the physicalspace where accuracy of the 3-D tracking system (which is typically partof the navigation system) is greater, may illustrate the available rangeof motion of the positioning system 208 and/or the optical imagingsystem 500, and/or may help to guide head and/or patient positioning.

The navigation system 205 may provide tools to the neurosurgeon that mayhelp to provide more relevant information to the surgeon, and may assistin improving performance and accuracy of port-based neurosurgicaloperations. Although described in the present disclosure in the contextof port-based neurosurgery (e.g. for removal of brain tumors and/or fortreatment of intracranial hemorrhages (ICH)), the navigation system 205may also be suitable for one or more of: brain biopsy,functional/deep-brain stimulation, catheter/shunt placement (in thebrain or elsewhere), open craniotomies, and/orendonasal/skull-based/ear-nose-throat (ENT) procedures, among others.The same navigation system 205 may be used for carrying out any or allof these procedures, with or without modification as appropriate.

In some examples, the tracking camera 213 may be part of any suitabletracking system. In some examples, the tracking camera 213 (and anyassociated tracking system that uses the tracking camera 213) may bereplaced with any suitable tracking system which may or may not usecamera-based tracking techniques. For example, a tracking system thatdoes not use the tracking camera 213, such as a radiofrequency trackingsystem, may be used with the navigation system 205.

FIG. 3 is a block diagram illustrating a control and processing system300 that may be used in the medical navigation system 205 shown in FIG.2 (e.g. as part of the equipment tower 207). As shown in FIG. 3, thecontrol and processing system 300 may include one or more processors302, a memory 304, a system bus 306, one or more input/output interfaces308, a communications interface 310, and storage device 312. The controland processing system 300 may interface with other external devices,such as a tracking system 321, data storage 342, and external user inputand output devices 344, which may include, for example, one or more of adisplay, keyboard, mouse, sensors attached to medical equipment, footpedal, and microphone and speaker. Data storage 342 may be any suitabledata storage device, such as a local or remote computing device (e.g. acomputer, hard drive, digital media device, or server) having a databasestored thereon. In the example shown in FIG. 3, data storage device 342includes identification data 350 for identifying one or more medicalinstruments 360 and configuration data 352 that associates customizedconfiguration parameters with one or more medical instruments 360. Thedata storage device 342 may also include preoperative image data 354and/or medical procedure planning data 356. Although the data storagedevice 342 is shown as a single device in FIG. 3, it will be understoodthat in other embodiments, the data storage device 342 may be providedas multiple storage devices.

The medical instruments 360 may be identifiable by the control andprocessing unit 300. The medical instruments 360 may be connected to andcontrolled by the control and processing unit 300, or the medicalinstruments 360 may be operated or otherwise employed independent of thecontrol and processing unit 300. The tracking system 321 may be employedto track one or more medical instruments 360 and spatially register theone or more tracked medical instruments to an intraoperative referenceframe. For example, the medical instruments 360 may include trackingmarkers such as tracking spheres that may be recognizable by thetracking camera 213. In one example, the tracking camera 213 may be aninfrared (IR) tracking camera. In another example, a sheath placed overa medical instrument 360 may be connected to and controlled by thecontrol and processing unit 300.

The control and processing unit 300 may also interface with a number ofconfigurable devices, and may intraoperatively reconfigure one or moreof such devices based on configuration parameters obtained fromconfiguration data 352. Examples of devices 320, as shown in FIG. 3,include one or more external imaging devices 322, one or moreillumination devices 324, the positioning system 208, the trackingcamera 213, one or more projection devices 328, and one or more displays206, 211.

Exemplary aspects of the disclosure can be implemented via theprocessor(s) 302 and/or memory 304. For example, the functionalitiesdescribed herein can be partially implemented via hardware logic in theprocessor 302 and partially using the instructions stored in the memory304, as one or more processing modules or engines 370. Exampleprocessing modules include, but are not limited to, a user interfaceengine 372, a tracking module 374, a motor controller 376, an imageprocessing engine 378, an image registration engine 380, a procedureplanning engine 382, a navigation engine 384, and a context analysismodule 386. While the example processing modules are shown separately inFIG. 3, in some examples the processing modules 370 may be stored in thememory 304 and the processing modules 370 may be collectively referredto as processing modules 370. In some examples, two or more modules 370may be used together to perform a function. Although depicted asseparate modules 370, the modules 370 may be embodied as a unified setof computer-readable instructions (e.g. stored in the memory 304) ratherthan distinct sets of instructions.

FIG. 4A illustrates use of an example optical imaging system 500,described further below, in a medical procedure. Although FIG. 4A showsthe optical imaging system 500 being used in the context of a navigationsystem environment 200 (e.g. using a navigation system as describedabove), the optical imaging system 500 may also be used outside of anavigation system environment.

An operator, typically a surgeon 201, may use the imaging system 500 toobserve the surgical site (e.g. to look down an access port). Theoptical imaging system 500 may be attached to a positioning system 208,such as a controllable and adjustable robotic arm. The position andorientation of the positioning system 208, imaging system 500, and/oraccess port may be tracked using a tracking system, such as describedfor the navigation system 205. The distance between the optical imagingsystem 500 (more specifically, the aperture of the optical imagingsystem 500) and the viewing target may be referred to as the workingdistance. The optical imaging system 500 may be designed to be used in apredefined range of working distance (e.g. in the range of between 15and 75 centimeters). It should be noted that, if the optical imagingsystem 500 is mounted on the positioning system 208, the actualavailable range of working distance may be dependent on both the workingdistance of the optical imaging system 500 as well as the workspace andkinematics of the positioning system 208. In some embodiments, theoptical imaging system 500 may include a manual release button that,when actuated, enables the optical imaging system to be positionedmanually. For example, the controller of the optical imaging system 500may be responsive to manual control input received via a user interface.

Reference is now made to FIG. 4B, which shows components of an exampleoptical imaging system 500. The optical imaging system 500 includes aprimary camera (or video-scope) 535. The primary camera 535 may be ahigh-definition (HD) camera that captures image data from the opticalassembly. The optical imaging system 500 may also include an opticalassembly 505. The optical assembly 505 may include optics (e.g. lenses,optical fibers, etc.) for focusing and zooming on the viewing target.The optical assembly 505 may include zoom optics 510 and focus optics515. Each of the zoom optics 510 and focus optics 515 are independentlymoveable within the optical assembly, in order to adjust the zoom andfocus, respectively. The optical assembly 505 may include an aperturewhich may be adjustable.

The optical imaging system 500 also includes a memory 550 and acontroller 530 coupled to the memory 550. The controller 530 maycomprise one or more processors (e.g. micro-processors), programmablelogic devices (e.g. field-programmable gate arrays, or FPGAs),application-specific integrated circuits (ASICs), or combinationsthereof. In at least some embodiments, the controller 530 is configuredto control operation of a zoom actuator and a focus actuator. Thecontroller 530 may receive control input indicating a desired zoomand/or focus and, in response to receiving the input, the controller 530may cause the zoom actuator and/or the focus actuator to move the zoomoptics 510 and focus optics 515, respectively.

The controller 530 is also configured to control operation of theprimary camera 535. The primary camera 535 may output camera data to thecontroller 530, which in turn transmits the data to an external systemfor viewing. The captured images can then be viewed on larger displaysand may be displayed together with other relevant information, such as awide-field view of the surgical site, navigation markers, etc.

In at least some embodiments, the primary camera 535, optical assembly505 (including the zoom optics 510 and focus optics 515), controller530, and memory 550 may all be housed within a single housing of theoptical imaging system 500. The housing may be provided with a frame onwhich trackable markers may be mounted to enable tracking by anavigation system. The optical imaging system 500 may be mountable on amoveable support structure, such as a positioning system (e.g. roboticarm) of a navigation system, a manually operated support arm, aceiling-mounted support, a moveable frame, or other support structure.In some embodiments, the optical imaging system 500 may include asupport connector, such as a mechanical coupling, to enable the opticalimaging system 500 to be mounted to and dismounted from the supportstructure.

FIGS. 5A-5E show different views of an example augmented optical imagingsystem 600. The augmented optical imaging system 600 includes one ormore of the components of the optical imaging system 500. In particular,the augmented optical imaging system 600 includes a primary camera 602for capturing an image of a target, zoom and focus optics, one or morelight sources 610, and a controller (not shown) for controllingoperation of the primary camera 602 and zoom, focus, and/or auxiliaryoptics.

In addition to these components, the augmented optical imaging system600 includes a 3-D optics module 630. The 3-D optics module 630 extendsthe functionalities of the optical imaging system 500. In particular,the 3-D optics module 630 comprises an add-on component to the opticalimaging system 500. In some embodiments, the 3-D optics module 630 maybe separable from the optical imaging system 500. For example, the 3-Doptics module 630 may be a separate device/module that can be mounted tothe optical imaging system 500 or components thereof, such as theprimary camera 602. In such embodiments, the optical imaging system 500may refer to that part of the augmented optical imaging system 600 whichis separate from the 3-D optics module 630. The 3-D optics module 630may enable the augmented optical imaging system 600 to obtain 3-Dinformation of a viewing target.

As shown in FIGS. 6A-6B and FIGS. 7C-7D, the 3-D optics module 630includes a secondary (e.g. outrigger) camera 604 for capturing an imageof a target and a pair of path folding mirrors 608A and 608B. Thesecondary camera 604 has a wide-field view, and may have at least one offixed zoom optics, fixed focus optics, or digital zoom capability. Thepath folding mirrors 608A and 608B are positioned in spaced relation toeach other. Specifically, the path folding mirrors 608A and 608B areangled with respect to each other such that they are disposed in anoptical path between a target being imaged by the secondary camera 604and a lens of the secondary camera 604. That is, light reflected off asurface of the imaged target traverses a path that includes the pathfolding mirrors 608A and 608B. The optical path of the secondary camera604 thus includes, at least, a first segment (S1) between the target anda reflective surface of a first path folding mirror 608A, a secondsegment (S2) between the reflective surface of the first path foldingmirror 608A and a reflective surface of a second path folding mirror608B, and a third segment (S3) between the reflective surface of thesecond path folding mirror 608B and a lens of the secondary camera 604.Accordingly, in at least some embodiments, the path folding mirrors 608Aand 608B are selectively positioned based on a position of the lens ofthe secondary camera 604. This optical path is shown in FIGS. 7C-7D.

The 3-D optics module 630 is configured to be connected to an opticalimaging system in order to augment the functionalities of the opticalimaging system. In particular, the 3-D optics module 630 may be affixeddirectly to an optical imaging system and secured thereto by a suitablefastening mechanism. As shown in FIG. 6B, the 3-D optics module 630defines a chamber/bore which is sized to receive the primary camera 602when the 3-D optics module 630 is secured to the optical imaging system.The optics of the primary camera 602 align with the opening 635 definedon the 3-D optics module 630. In some embodiments, the primary camera602 may extend through the opening 635 when the 3-D optics module 630 issecured to the optical imaging system.

Returning to FIGS. 5A-5E, a controller of the augmented optical imagingsystem 600 is configured to receive a first image from the primarycamera 602 and a second image from the secondary camera 604. Forexample, the primary camera 602 and secondary camera 604 may acquirereal-time camera data (e.g. videos, images, etc.) depicting a target. Inat least some embodiments, the primary camera 602 and the secondarycamera 604 are positioned such that the optical axis of the primarycamera 602 is co-planar with the optical axis of the secondary camera604. The primary camera 602 may be offset both vertically andhorizontally relative to the secondary camera 604. In some embodiments,the primary camera 602 and the secondary camera 604 may be offset onlyhorizontally.

FIG. 8 shows the augmented optical imaging system 600 mounted to apositioning system 208 (e.g. a robotic arm) of a navigation system. Theaugmented optical imaging system 600 is shown with a housing thatencloses the zoom and focus optics, the primary camera 602, thesecondary camera 604, and a pair of path folding mirrors 608A and 608B.

Furthermore, FIG. 8 shows the secondary camera 604 being angled withrespect to the primary camera 602. In particular, the primary camera 602is positioned substantially vertically within the housing of theaugmented optical imaging system while the secondary camera 604 ispositioned at an angle with respect to the vertical. The path foldingmirrors 608A and 608B are disposed in the 3-D optics module 630 suchthat the optical path for the secondary camera 604 does not intersectthe optical path for the primary camera 602. Specifically, the pathfolding mirrors 608A and 608B are positioned so that the optical pathfor the secondary camera 604 does not obstruct the substantiallyvertical line of sight of the primary camera 602.

FIG. 9 is a perspective view of another example augmented opticalimaging system 900. The augmented optical imaging system 900 may beincorporated into a digital microscope system, and more generally, amedical navigation system. The augmented optical imaging system 900includes an optical imaging system 950 and a 3-D optics module 930. Theoptical imaging system 950 includes, at least, a primary camera 902 forimaging a target and one or more light sources 910. The 3-D opticsmodule 930 may be integral to the optical imaging system 950, or it maybe a separable add-on component which can be secured to the opticalimaging system 950. The 3-D optics module 930 includes a secondarycamera 904 and a single path folding mirror 908. As shown in FIG. 9, theposition of the path folding mirror 908 may be variable. For example, insome embodiments, a relative angle of the reflective surface of the pathfolding mirror 908 with respect to a lens of the secondary camera 904 isadjustable, either manually or via a control input. An actuatorassociated with the path folding mirror 908 may be controlled by acontroller (not shown) of the augmented optical imaging system 900. Inother embodiments (not shown), the actuator may be manually moved toconfigure the relative angle.

In the example of FIG. 9, the secondary camera 904 is positionedsubstantially orthogonal to the primary camera 902. In particular, theprimary camera 902 is directed vertically downward, while the secondarycamera 904 is directed substantially horizontally. The 3-D optics module930 may include a plate 920 which can be secured to the optical imagingsystem 950. The plate 920 is generally planar and elongate, and isdisposed generally orthogonal to the optical imaging system 950. Thatis, the plate 920 is substantially horizontal when secured to theoptical imaging system 950. As shown in FIG. 9, the secondary camera 904may be affixed to the plate 920.

The path folding mirror 908 is disposed in an optical path between atarget being imaged and a lens of the secondary camera 904. That is, anoptical path of the secondary camera 904 traverses a path defined by afirst segment between the target and a reflective surface of the pathfolding mirror 908 and a second segment between the reflective surfaceof the path folding mirror 908 and a lens of the secondary camera 904.The path folding mirror 908 is located on the 3-D optics module 930 suchthat it does not obstruct a (vertical) line of sight of the primarycamera 902. That is, the path folding mirror 908 does not interfere withan optical path of the primary camera 902.

Reference is now made to FIG. 10 which shows, in flowchart form, anexample method 1000 for generating a 3-D image of a target using anaugmented optical imaging system. The method 1000 may be implemented ina digital microscope system. For example, the method 1000 may beimplemented by a controller of an augmented optical imaging systemintegrated into a digital microscope, or similar processing unit forcontrolling operations of cameras of an augmented optical imagingsystem.

In operation 1002, the controller receives a first image from theprimary camera, and in operation 1004, the controller receives a secondimage from the secondary camera. The controller then applies an imagetransform to one of the first image and the second image, in operation1006. In at least some embodiments, the image transform is a homographictransform. In particular, the image transform may implement a homographyused for image rectification. With known relative camera positions, thehomography warps one of the images such that the first and second imagesappear as if they have been taken with only a horizontal displacement,thereby simplifying the stereo matching process in generating 3-Dvisualization of the target. In some embodiments, the controller may beconfigured to determine a working distance (i.e. stand-off distance)between the target and an aperture of the optical imaging system (oropening for the cameras' lines of sight) and determine the imagetransform to be applied to the one of the images based on the workingdistance.

The determination of the homographic transform to apply in operation1006 may be done based on an interpolation scheme. That is, thecontroller may be configured to interpolate between two or morecalibration homographies. Further, the controller may search a range ofinterpolated homographies and determine a “best” homography transform toapply to images from the secondary camera in generating 3-Dvisualizations. This may be done by, for example, applying each of aplurality of homographic transforms (i.e. warping) to images from thesecondary camera and computing a metric that represents imagecorrespondence between the warped images and the corresponding imagesfrom the primary camera. The controller may take, as inputs, a transformof an image from the secondary camera and a corresponding (i.e. capturedsubstantially concurrently) image from the primary camera, and output avalue for a relevant metric. A homography that produces an optical valuefor the metric in question can be selected as the “best” homography.

Various different metrics may be suitably employed by the controller inthe image comparisons. The metric may, for example, comprisecorrelation, mutual information, difference of squares, etc. Thecomputation of the metric may be done for the entire range ofinterpolated homography transforms under investigation. Depending on themetric that is used, the controller may look for either a local maximumvalue or a local minimum value in identifying the transform that resultsin highest image correspondence, or best match. For example, if adifference of squares metric is used, the controller would look for thehomography producing the lowest value for the metric from among theinterpolated transforms. As another example, if image correlation is themetric used, a homography that produces the highest value for the metricmay be selected as the best homography.

In operation 1008, the controller combines the transformed image and theother one of the images to generate a stereoscopic image of the target.In at least some embodiments, the controller may perform calibration ofthe zoom of the primary and secondary cameras prior to generating thestereoscopic image. For example, if the augmented optical imaging systemhas been moved to a significant degree or a predefined period of timehas elapsed since last calibration of the cameras, the controller may beconfigured to automatically calibrate zoom. In some embodiments, theaugmented optical imaging system may auto-calibrate for a plurality ofpredefined stand-off distances.

Returning to FIG. 4A, the navigation system 200 may be adapted toprovide 3-D information of a viewing target. Specifically, thenavigation system 200 may incorporate a 3-D visualization setup for useduring a medical procedure. As shown in FIG. 5A, the 3-D visualizationsetup may include an optical imaging system 500 that includes a primarycamera, a secondary camera, and at least one path folding mirror. In atleast some embodiments, the primary camera (which may be the opticalhead of the optical imaging system 500), the secondary camera, and theat least one path folding mirror may be housed within a single housing.The optical imaging system 500 may be connected to a positioning system208, such as a mechanical arm or stand, which is controllable,adjustable, and moveable. The optical imaging system 500 may be mountedto the positioning system 208 such that the positioning system 208 canposition and orient the optical imaging system 500.

Operation of the optical imaging system 500 may be controlled by aprocessing unit of the optical imaging system 500 or the navigationsystem 200. The processing unit is configured to generate 3-Dstereoscopic images of a viewing target, based on images acquired by theprimary and secondary cameras. For example, the processing unit mayimplement a method for generating 3-D information, such as method 1000of FIG. 10. The processing unit may also be configured to implement acalibration module for calibrating images from the cameras. Thecalibration module may, for example, determine a current position andorientation of the cameras of the optical imaging system 500. Thecalibration module may also determine transforms (e.g. homographies) toapply to images of the cameras for providing 3-D visualization of theviewing target.

The optical imaging system 500 may transmit data to the controller or toan external system, such as an external work station. The image dataacquired by the optical imaging system 500 is used to generate 3-Dstereoscopic images of the viewing target. The stereoscopic imageinformation may be displayed, for example, on a 3-D display device 230(e.g. 3-D monitor) that is viewable using 3-D glasses donned by thesurgeon 201 during a procedure. The 3-D information may also be usefulfor an augmented reality (AR) display. For example, an AR display systemmay use information acquired by the navigation system 200 and overlay3-D images of a target specimen on a real-time image captured by thecameras.

The various embodiments presented above are merely examples and are inno way meant to limit the scope of this application. Variations of theinnovations described herein will be apparent to persons of ordinaryskill in the art, such variations being within the intended scope of thepresent application. In particular, features from one or more of theabove-described example embodiments may be selected to createalternative example embodiments including a sub-combination of featureswhich may not be explicitly described above. In addition, features fromone or more of the above-described example embodiments may be selectedand combined to create alternative example embodiments including acombination of features which may not be explicitly described above.Features suitable for such combinations and sub-combinations would bereadily apparent to persons skilled in the art upon review of thepresent application as a whole. The subject matter described herein andin the recited claims intends to cover and embrace all suitable changesin technology.

The invention claimed is:
 1. An optical imaging system for imaging atarget during a medical procedure, the system comprising: a first camerafor capturing a first image of the target; a second wide-field camerafor capturing a second image of the target; at least one path foldingmirror disposed in an optical path between the target and a lens of thesecond camera; and a processing unit for receiving the first image andthe second image, the processing unit being configured to: apply animage transform to one of the first image and the second image; andcombine the transformed image with the other one of the images toproduce a stereoscopic image of the target, wherein the at least onepath folding mirror comprises a first mirror and a second mirror thatare selectively positioned based on a position of the lens of the secondcamera, the first mirror and the second mirror being angled with respectto each other.
 2. The optical imaging system of claim 1, wherein thefirst camera, the second camera, and the at least one path foldingmirror are housed within a single housing.
 3. The optical imaging systemof claim 1, wherein the second camera and the at least one path foldingmirror are included in an add-on module for mounting to the firstcamera.
 4. The optical imaging system of claim 1, wherein the firstmirror is selectively positioned and angled with respect to the targetso as to reflect an image of the target to the second mirror, and thesecond mirror is selectively positioned and angled so as to reflect theimage of the target from the first mirror to the lens of the secondcamera.
 5. The optical imaging system of claim 1, wherein the firstcamera and the second camera are positioned such that an optical axis ofthe first camera is co-planar with the optical axis of the secondcamera.
 6. The optical imaging system of claim 1, wherein the imagetransform is a homographic transform.
 7. The optical imaging system ofclaim 1, wherein the processing unit is further configured to: determinea working distance between the target and an aperture of the opticalimaging system; and determine the image transform based on the workingdistance.
 8. The optical imaging system of claim 1, wherein the opticalimaging system is configured to be mountable onto a moveable supportstructure.
 9. The optical imaging system of claim 8, wherein the opticalimaging system further comprises a support connector to enable theoptical imaging system to be removably mounted onto the moveable supportstructure.
 10. The optical imaging system of claim 9, wherein themoveable support structure comprises one of a robotic arm, amanually-operated support arm, or a moveable support frame.
 11. Theoptical imaging system of claim 10, further comprising a manual releasebutton that, when actuated, enables the optical imaging system to bepositioned manually.
 12. The optical imaging system of claim 11, whereinthe processing unit is responsive to control input received via a userinterface.
 13. The optical imaging system of claim 1, further comprisingone or more light sources.
 14. The optical imaging system of claim 1,wherein the second camera has at least one of fixed zoom optics or fixedfocus optics.
 15. The optical imaging system of claim 1, wherein thesecond camera is fixedly coupled to the first camera.
 16. A method ofgenerating a stereoscopic image of a target in a medical procedure usingan optical imaging system, the method comprising: receiving, from afirst camera of the optical imaging system, a first image of the target;receiving, from a second camera of the optical imaging system, a secondimage of the target; applying an image transform to one of the firstimage and the second image wherein the image transform is a firsthomographic transform; selecting the first homographic transform from aplurality of homographic transforms, wherein the selecting comprises:for each of the plurality of homographic transforms: applying thehomographic transform to the second image; and computing an imagecorrespondence metric between the transformed second image and the firstcamera, and selecting the homographic transform that is associated withan optimal value of the image correspondence metric from the pluralityof homographic transforms as the first homographic transform; andcombining the transformed image with the other one of the images toproduce the stereoscopic image of the target.
 17. The method of claim16, further comprising: determining a working distance between thetarget and an aperture of the optical imaging system; and determiningthe image transform based on the working distance.